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Abstract 

This  thesis  examined  the  robustness  of  methods  used  to 
compute  estimates  at  completion  of  contract  costs.  The 
four  techniques  examined  were  a  subset  of  those  used  by  the 
Cost  Performance  Report  Analysis  computer  program  to  derive 
estimates  at  completion. 

Monte  Carlo  simulation  methods  were  used  to  generate 
the  required  data  values  for  18  simulated  populations 
representing  a  variety  of  possible  cost  and  schedule 
overrun  scenarios.  A  statistical  evaluation  of  the 
robustness  of  the  four  estimate  at  completion  techniques 
was  performed  using  samples  from  each  of  the  18  simulated 
populations . 

The  results  of  this  analysis  indicate  that,  for 
practical  purposes,  none  of  the  selected  estimate  at 
completion  techniques  are  robust  for  the  cost  and  schedule 
overrun  scenarios  that  were  investigated.  The  tendency  of 
each  of  the  estimate  at  completion  models  was  to 
underestimate  the  final  contract  cost.  The  estimate  at 
completion  models  incorporating  mathematical  smoothing 
techniques  (i.e.  weightings  or  moving  averages)  provided 
relatively  stable  results  across  all  of  the  populations; 
however,  they  were  unable  to  consistently  predict  a 
contract's  completion  cost  within  30  percent  of  the  actual 


ROBUSTNESS  IN  COST  PERFORMANCE  MODELS 


I .  Introduction 


General  Issue 

Department  of  Defense  (DOD)  Instruction  7000.2, 
Performance  Measurement  for  Selected  Acquisitions, 
established  the  Cost/Schedule  Control  Systems  Criteria 
(C/SCSC)  as  a  method  of  assessing  contractor  performance  on 
major  systems  acquisitions  (19:36).  The  primary  objectives 
of  C/SCSC  are  to  ensure  DOD  contractors  use  effective 
internal  management  control  systems  and  to  provide  the 
government  with  timely  and  auditable  data  for  monitoring 
contract  progress  (1:45).  Air  Force  Systems  Command 
Pamphlet  173-5,  Cost  Analysis  [ : 1  Cost/Schedule  Control 
Systems  Criteria  Joint  Implementation  Guide,  identifies  the 
minimum  requirements  for  contractors'  management  control 
systems : 

1.  Realistic  budgets  for  work  scheduled  within 
responsibility  assignments  (6:5). 

2.  Accurate  accumulation  of  costs  related  to  progress 
of  the  planned  work  (6:5). 

3.  Comparison  between  the  actual  resources  applied 
and  the  estimated  resources  planned  for  specific  work 
assignments  (6:5). 

4.  Preparation  of  reliable  estimates  of  costs  to 
complete  remaining  work  (6:5). 


5.  Support  of  an  overall  capability  for  managers  to 
analyze  available  information  to  identify  problem  areas 
in  sufficient  time  to  take  remedial  action  (6:5). 

The  C/SCSC  have  no  explicit  reporting  requirements; 

however,  summary  data  generated  by  the  contractor's 

internal  management  control  system  are  reported  to  the 

government  program  office  through  the  Cost  Performance 

Report  (CPR)  (1:46).  A  required  data  item  is  the  Estimate 

at  Completion  (EAC)  which  is  computed  for  each  element  in 

the  Work  Breakdown  Structure  and  each  major  functional 

category  within  the  organization  (5:ii).  The  EAC  is  a 

"continually  revised  value  which  reflects  the  contractor's 

estimate  to  complete  the  physical  requirements  of  the 

contract"  (5:ii). 

The  EAC  is  an  important  measure  of  the  contractor's 
performance  and  a  primary  input  into  periodic  program 
status  reports  sent  to  the  major  acquisition  commands,  the 
Office  of  the  Secretary  of  Defense,  and  the  Congress 
(1:47).  Therefore,  it  is  particularly  important  to  verify 
the  methods  and  database  used  by  the  contractor  to 
calculate  the  EAC. 

Specific  Problem 

The  Department  of  Defense  "does  not  have  the  resources 
to  examine  the  internal  plans  and  resource  allocations  of 
the  contractor;  it  must  make  its  predictions  (EACsl  on  the 
basis  of  contractor  performance  on  work  already  completed" 
(4:2).  As  a  result,  a  number  of  automated  techniques  have 


been  developed  to  generate  BACs  on  the  basis  of  contractor 
performance  to  date.  Holeman  Identifies  the  major  problem 
in  applying  these  techniques;  "there  does  not  appear  to 
have  been  any  attempt  to  conduct  in-depth  or  Independent 
reviews  of  any  of  these  automated  approaches  to  offer 
unbiased  and  expert  guidance  on  when  and  where  they  should 
or  should  not  be  used"  (11:40).  The  choice  of  an 
appropriate  technique  is  further  complicated  by  the  fact 
that  several  "of  these  techniques  could  be  considered 
analytically  sophisticated  and  many  analysts  in  program 
offices  would  probably  have  difficulty  in  fully 
understanding  their  supporting  documentation"  (11:40). 
Consequently,  analysts  may  apply  inappropriate  techniques 
to  generate  EACs  believed  to  be  accurate  because  they  have 
been  generated  by  a  computer  to  eight  decimal  places;  when, 
in  actuality,  they  may  be  quite  unrealistic. 

This  thesis  evaluates  the  robustness  of  four 
techniques,  or  models,  for  generating  EACs.  Estimates  can 
be  point  or  Interval  estimates;  this  research  is  Interested 
in  the  robustness  of  point  estimates--3peclf ically,  point 
estimates  of  contract  completion  costs.  A  point  estimate 
may  be  considered  robust  if  it  produces  acceptable  results 
for  a  variety  of  conditions  or  situations.  The  definition 
of  "robust"  depends  on  who  is  defining  it  and,  thus,  is 
necessarily  subjective.  For  example,  a  point  estimate  of  a 
contract's  cost  may  be  considered  robust  by  one  analyst  if 


it  is  within  10  percent  o£  the  final  contract  cost  over  a 
wide  range  of  conditions;  another  analyst  may  only  require 
the  estimate  to  be  within  30  percent  of  the  final  cost  to 
be  considered  robust  (14:1).  A  robust  model  would  be 
useful  because  it  would  allow  analysts  to  achieve  adequate 
results  over  a  wide  range  of  situations. 


Definition  of  Key  Terms 

The  following  terms  are  used  throughout  this  thesis 
and  are  presented  to  aid  the  reader's  understanding. 

1.  Actual  Cost  of  Work  Performed  (ACWP).  The  costs 
actually  incurred  and  recorded  in  accomplishing  the 
work  performed  within  a  given  time  period  (6:5). 

2.  Budget  at  Completion  (BAC).  The  budgetary  goal 
for  total  amount  of  work  required  on  the  contract 
(5:5) . 

3.  Budgeted  Cost  for  Work  Performed  (BCWP).  The  sum 
of  the  budgets  for  completed  work  packages  and 
completed  portions  of  open  work  packages,  plus  the 
appropriate  portion  of  the  budgets  for  level  of 
effort  and  apportioned  effort  (6:6). 

4.  Budgeted  Cost  for  Work  Remaining  (BCWR).  The 
difference  between  the  budgetary  goal  (BAC)  and  the 
sum  of  the  budgets  for  work  completed  to  date 
(cumulative  BCWP). 

5.  Budgeted  Cost  for  Work  Scheduled  (BCWS).  The  sum 
of  the  budgets  for  all  work  packages,  planning 
packages,  etc.,  scheduled  to  be  accomplished 
(including  in-process  work  packages),  plus  the  amount 
of  level  of  effort  and  apportioned  effort  scheduled 
to  be  accomplished  within  a  given  time  period  (6:6). 

6.  Estimated  Cost  at  Completion  or  Estimate  at 
Completion  (EAC).  Actual  di.ect  costs,  plus  indirect 
costs  allocable  to  the  contract,  plus  the  estimate  of 
costs  (direct  and  indirect)  for  authorized  work 
remaining  (6:6). 


7.  Performance  Measurement  Baseline.  The 
time-phased  budget  plan  against  which  contract 
performance  is  measured.  It  equals  the  total 
allocated  budget  less  management  reserve  (6:6). 

8.  Work  Packages.  Detailed  short-span  jobs,  or 
material  items,  identified  by  the  contractor  for 
accomplishing  work  required  to  complete  the  contract 
(6:6)  . 

Description  of  EAC  Formulas 

The  EAC  formulas  used  in  this  research  are  four  of  the 
automated  methods  for  generating  EACs  employed  by  the  Air 
Force  Wright  Aeronautical  Laboratories  (AFWAL)  at 
Wr ight-Patterson  Air  Force  Base,  Ohio.  These  methods  were 
selected  for  study  to  further  the  research  previously 
accomplished  by  Captain  James  Price  and  documented  in  his 
thesis.  An  Evaluation  of  CPRA  Estimate  at  Completion 
Techniques  Based  Upon  AFWAL  Cost/Schedule  Control  System 
Cr iter  la  Data . 

Formula  Convention .  In  the  EAC  formulas  and  several 
variance  and  index  equations,  it  is  often  necessary  to 
reference  two  data  points  with  the  same  variable  name  (for 
example,  the  value  of  BCWP  for  the  months  of  June  and 
July).  When  this  occurs,  the  following  convention  will  be 
used.  (This  convention  is  similar  to  that  used  by  Captain 
Price.)  The  data  point  will  retain  its  variable  name  (i.e. 
"BCWP"),  but  an  indicator  of  its  position  in  the  array  of 
data  points  also  will  be  'attached.  The  last  month  for 
which  data  is  available  will  have  the  Indicator,  "(CUM)," 


added.  Preceding  data  points  in  the  array  will  be 


described  with  the  Indicator^  "(CUM-n),"  where  "n"  Is  the 
number  of  months  prior  to  the  last  data  point.  An  example 
should  make  this  clear.  If  BCVP  data  is  available  for 
August  through  November,  the  cumulative  data  point 
associated  with  November  would  be  Identified  as 
”BCWP(CUM)."  Similarly,  the  cumulative  data  points  for 
August,  September,  and  October,  would  be  described  by 
"BCWP(CUM-3),"  "BCWP(CUM-2),“  and  "BCWP ( CUM-1 ) , ” 
respectively. 

Variances  and  Indices .  A  number  of  variances  and 
indices  are  used  by  the  EAC  formulas  to  calculate  estimates 
at  completion.  These  equations  are  discussed  below  and 
later  will  be  incorporated  into  the  BAG  formulas. 

COST  VAR  %  »  [(BCWP(CUM)  -  ACWP ( CUM )) /BCWP ( CUM ) J 

*  100  (1) 

This  variance  measures  whether  work  accomplished 
to  date  is  "over”  or  "under"  budget.  A  negative  cost 
variance  percentage  indicates  actual  work 
accomplished  is  costing  this  percentage  more  than 
planned  (cost  overrun).  Similarly,  a  positive  value 
indicates  actual  work  accomplished  is  costing  this 
percentage  less  than  planned  (cost  underrun). 


CUR  MON  CPI  =  (BCWP(CUM)  -  BCWP ( CUM-1 )) / 

(ACWP (CUM)  -  ACWP ( CUM-1) )  (2) 

This  Is  a  Cost/Performance  Index  for  the  current 
month  which  indicates  how  much  planned  work  the 
contractor  performed  for  each  dollar  that  was 
actually  spent.  For  example,  if  the  value  for  CUR 
MON  CPI  was  .75,  this  means  that  for  every  dollar 
actually  spent  in  the  current  month,  only  $.75  worth 
of  work  was  performed. 


-6- 


THREE  MON  CPI 


(BCWP(CUM)  -  BCWP(CUM-3) )/ 


(ACWP(CUM)  -  ACWP(CUM-3))  (3) 

Similar  to  CUR  MON  CPI  except  this  Index  Is  a 
measure  of  the  contractor's  cost  performance  for  the 
last  three  months. 


CUM  CPI  =  BCWP(CUM)/ACWP(CUM)  (4) 

This  Cost/Performance  Index  is  a  measure  of  the 
contractor's  cost  performance  from  the  start  of  the 
contract  through  the  latest  month  for  which  data  Is 
available . 


SCH  VAR  %  =  I(BCWP(CUM)  -  BCWS ( CUM ) ) /BCWS ( CUM ) 1 

*  100  (5) 

This  variance  measures  whether  work  accomplished 
to  date  Is  "behind"  or  "ahead"  of  schedule.  A 
negative  schedule  variance  percentage  Indicates  this 
percentage  less  work  was  performed  than  scheduled 
(behind  schedule).  Similarly,  a  positive  value 
Indicates  this  percentage  more  work  was  performed 
than  scheduled  (ahead  of  schedule). 


EAC  Formulas .  Four  of  the  formulas  used  by  AFWAL's 
Cost  Performance  Report  Analysis  (CPRA)  computer  program  to 
generate  estimates  at  completion  have  been  reproduced  from 
Capt  Price's  thesis  and  are  shown  below.  Throughout  the 
remainder  of  this  thesis,  these  formulas  will  be  referenced 
by  the  Indicator,  "EACn,"  where  "n"  Is  a  number  from  1  to  4 
corresponding  to  a  particular  EAC  technique. 


EACl  =  ACWP(CUM)  +  ( BCWR ( CUM ) /CUR  MON  CPI)  (6) 


where 


BCWR ( CUM ) 


BAC  -  BCWP(CUM) 


(7) 


This  formula  takes  into  account  all  cost 
overruns  that  have  occurred  to  date  In  the  form  of 
ACWP(CUM),  but  assumes  such  overruns  are 
nonrecurring.  It  also  assumes  the  remaining  work 
will  be  performed  with  the  same  budget  efficiency  as 
the  most  recent  month  (CUR  MON  CPI). 


EAC2  =  ACWP(CUM)  +  ( BCWR ( CUM ) /THREE  MON  CPI)  (8) 

This  method  accounts  for  any  cost  overruns  to 
date  (ACVP(CUM))  and  assumes  the  contractor  will 
perform  the  remaining  work  with  the  budget  efficiency 
displayed  during  the  last  three  months  (THREE  MON 
CPI )  . 


EAC3  =  ACWP(CUM)  +  ( BCWR ( CUM ) /CUM  CPI)  (9) 

This  calculation  also  accounts  for  all  cost 
overruns  to  date  (ACVP(CUM)),  but  assumes  the 
remaining  work  will  be  performed  with  the  same  budget 
efficiency  exhibited  since  the  start  of  the  contract 
(CUM  CPI  )  . 


where 


BAC4  »  ACWP(CUM)  +  ETC 


(10) 


ETC  =  (100  -  (COST  VAR  %)  -  .75  •  ( SCH  VAR  \)1 

*  BCWR(CUM)/100  (11) 

This  technique  includes  the  cost  overruns  to 
date  (ACWP(CUM)),  but  also  assumes  the  cost  variance 
percentage  will  remain  constant  for  the  remainder  of 
the  program.  Additionally,  a  penalty  of  75\  of  the 
schedule  variance  percentage  is  used  to  compute 
catch-up.  (It  is  believed  that  a  typographical  error 
exists  in  Price's  thesis  for  this  equation--in  order 
to  assess  a  "penalty"  of  75%  of  the  schedule  variance 
percentage,  the  equation  must  be  changed  to  read 
"-.75,"  rather  than  "+  .75";  therefore,  the  equation 
was  altered . ) 


Scope  and  Limitations 

This  thesis  conducted  an  evaluation  of  the  robustness 


of  four  methods  for  calculating  program  estimates  at 


completion  under  a  variety  of  simulated  program  scenarios. 
These  four  models  represent  only  a  small  subset  of  the  BAC 
techniques  available  to  analysts  and  It  should  not  be 
inferred  that  these  are  the  "best"  techniques  for  computing 
BACs .  Additionally,  each  of  these  formulas  compute  EACs 
based  on  the  contractor's  past  performance.  As  such,  the 
ability  of  the  formulas  to  accurately  predict  a  program's 
final  cost  is  limited  if  the  program  Is  new  and  little  data 
is  available.  Also,  this  research  simulated  only  a  number 
populations  representing  a  variety  of  possible  cost  overrun 
scenarios.  These  populations  are,  by  no  means,  exhaustive 
of  the  possible  scenarios  that  could  exist  in  the  "real" 
acquisition  environment;  but,  it  is  believed  there  is 
sufficient  variability  in  the  scenarios  to  allow 
conclusions  to  be  drawn  about  the  robustness  of  the  various 
BAC  techniques  examined. 

Research  Objective 

The  research  hypothesis  is  that  statistical  evaluation 
of  each  of  the  four  selected  models  for  generating 
estimates  at  completion  may  suggest  a  robust  technique. 

The  research  objective  was  to  determlr.e  the  robustness  of 
each  of  these  techniques  under  a  number  of  simulated  cost 


overrun  scenarios. 


Research  Questions .  Answers  to  the  following  research 
questions  will  be  determined  during  this  research. 

1.  What  are  the  statistical  moments  associated  with 
the  EAC  distributions  generated  by  the  four 
techniques  under  various  simulated  cost  and  schedule 
overrun  situations? 

2.  Are  there  significant  differences  in  the  mean  and 
variance  of  the  EAC  distributions  generated  by 
different  EAC  models  for  a  given  cost  and  schedule 
overrun  scenario?  by  the  same  EAC  model  for  various 
cost  and  schedule  overrun  scenarios? 

3.  Which  EAC  technique  is  the  most  robust  for  given 
cost  and  schedule  overrun  scenarios? 


LITERATURE  REVIEW 

The  literature  review  presents  background  information 
on  topics  relevant  to  this  research  project.  Specifically, 
the  topics  discussed  will  be  1)  Cost/Schedule  Control 
Systems  Criteria,  2)  estimates  at  completion  of  contract 
costs,  and  3)  Monte  Carlo  simulation. 

Cost/Schedule  Control  Systems  Criteria 

Department  of  Defense  Instruction  (DODI)  7000.2, 
Performance  Measurement  for  Selected  Acquisitions,  first 
Issued  in  1967,  requires  incorporation  of  the  Cost/Schedule 
Control  Systems  Criteria  (C/SCSC)  in  all  major  systems 
acquisitions  (excluding  fixed  price  contracts)  (1:45; 
17:11).  Major  systems  acquisitions  are  defined  by  DOD 
Directive  5000.1  and  office  of  Management  and  Budget 
Circular  A-109  as  programs  with  estimated  research, 
development,  and  test  (RDT&E)  costs  in  excess  of  $200 
million  or  estimated  production  costs  in  excess  of  $1 
billion  (12:1).  However,  these  are  not  "hard" 
thresholds-- the  DOD  acquisition  command  may  authorize 
deviations  from  these  thresholds  if  it  is  deemed 
appropriate.  "There  may  be  just  cause  for  applying  the 
criteria  to  other  programs  or  specific  contracts  such  as 
programs  designated  by  the  military  departments  as  major 
system  programs  of  national  interest  to  DOD  or  the  Congress 


because  of  their  critical  nature,  risk  involved  or  national 


security"  (8:44).  Fixed  price  contracts  are  excluded 
because  the  contractor  is  assuming  the  cost  and  schedule 
risks;  1£  the  contractor's  internal  management  system  is 
inefficient^  overruns  will  occur  and  profits  will  be  lost. 
Conversely,  in  cost  sharing  type  contracts,  more  efficient 
management  by  the  contractor  translates  into  reduced  costs 
to  the  government  (19:36-37). 

Objectives .  Abba  states  the  objectives  of  the  C/SCSC 

are : 

-  For  contractors  to  use  effective  internal  cost  and 
schedule  management  control  systems,  and 

-  For  the  government  to  be  able  to  rely  on  timely  and 
auditable  data  produced  by  those  systems  for 
determining  product-oriented  contract  status  (1:45). 

Both  objectives  could  be  ac?ompllshed  by  a  variety  of 

management  control  systems.  An  important  aspect  of  the 

C/SCSC  is  that  it  Is  not  a  system — it  does  not  impose  a 

particular  internal  management  system  on  the  contractor. 

It  is  a  set  of  criteria  or  standards  that  define  an 

acceptable  contractor  management  control  system  (1:45; 

2:32;  18:36).  The  concept  of  the  C/SCSC  is  that  if  a 

contractor  already  operates  an  adequate  cost  and  schedule 

management  control  system,  the  program  office  should  be 

able  to  extract  the  required  information  regarding  project 

status  (9:15).  Incorporating  the  C/SCSC  is  not  intended  to 

replace  or  duplicate  a  contractor's  existing 

system--changes  "are  required  only  to  the  extent  that  it 

does  not  meet  the  criteria"  (1:45). 


vu  vwii^i  «u  vun  in  w*  >  wwunnmmn* 


Requirements .  Department  of  Defense  instruction 
7000.2  and  the  C/SCSC  joint  Implementation  guide  specify 
the  requirements  a  contractor's  internal  management  system 
must  possess  to  be  acceptable  to  the  DOD — the  "what"  and 
not  the  "how."  "Briefly  stated,  C/SCSC  requires  the 
contractor  to  define  the  work  required  to  meet  contract 
objectives,  assign  the  work  to  specifically  identified 
organizational  elements,  establish  Internal  schedules  and 
budgets,  and  periodically  compare  actual  cost  and  schedule 
performance  against  the  planned  budgets  and  schedules" 
(19:37).  The  guide  outlines  In  five  broad  categories  35 
criteria  that  must  be  met  by  the  contractor's  system.  The 
five  categories  Include  organization,  planning  and 
budgeting,  accounting,  analysis,  and  revision  and  access  to 
data  (2:33;  8:16-17). 

Organizational  Criteria.  These  criteria  require 
the  contractor  to  precisely  define  all  contractual  work. 
This  is  accomplished  by  using  a  work  breakdown  structure 
(WBS)  which  breaks  up  the  total  effort  into  specific  units 
of  work  or  work  packages.  Additionally,  the  contractor 
must  clearly  identify  the  appropriate  functional 
organizational  level  or  manager  for  each  of  the  work  units. 
This  level  becomes  a  management  control  point  where  the 
contractor  will  establish  cost  accounts  to  allow  cost 
control  and  monitor  performance  (9:16;  15:29-30). 
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require  the  contractor  to  systematically  schedule,  budget, 
and  authorize  the  total  contractual  effort.  The  contractor 
must  devise  a  master  schedule  that  identifies  key 
milestones  and  work  interdependencies,  a  budgeting  system 
that  allocates  the  total  budget  to  each  of  the  WBS  elements 
and  functional  organizations,  and  a  work  authorization 
system  that  assigns  defined  tasks  to  the  appropriate 
functional  organizations.  A  critical  element  of  these 
criteria  is  the  development  of  a  time-phased  budget  known 
as  the  performance  measurement  baseline  (9:16;  15:30). 

The  authorized  work  is  scheduled  "in  time-phased  'planned 
value'  increments .. .as  work  is  accomplished,  it  is  'earned' 
on  the  same  budget  dollar  basis"  (1:45).  Comparing  the 
earned  and  planned  values  provides  a  measure  of  contract 
performance  in  the  form  of  a  schedule  variance. 

Accounting  Criteria .  These  criteria  require  the 
contractor  to  accumulate  the  actual  costs,  direct  and 
indirect,  of  all  the  accomplished  work.  Direct  costs  must 
be  identified  at  the  cost  account  level  for  summarization 
to  higher  management,  but  indirect  costs  may  be  allocated 
at  some  higher  level.  However  the  indirect  costs  are 
allocated,  cost  data  must  be  summarized  by  both  the  work 
breakdown  structure  and  functional  organization.  A 
comparison  of  actual  cost  with  earned  value  yields  a  cost 


variance  which  Indicates  whether  or  not  the  contractor  Is 
under-  or  overrunning  the  budget  (1:45;  8:16;  15:30). 

Analysis  Criteria .  These  criteria  describe  the 
"characteristics  that  contractors'  systems  must  possess, 
and  specify  the  type  of  data  that  should  be  derived  from 
the  contractors'  systems  to  adequately  measure  and  address 
performance"  (9:16).  A  number  of  key  terms  are  introduced 
by  the  analysis  criteria,  some  of  which  include: 

1.  Actual  Cost  of  Work  Performed  (ACWP).  The  costs 
actually  Incurred  by  the  contractor  in  performing  the 
work  during  a  given  time  period  (6:5). 

2.  Budget  at  Completion  (BAC).  The  budgeted  amount 
for  the  total  work  to  be  accomplished  under  the 
contract  (5:5). 

3.  Budgeted  Cost  for  Work  Performed  (BCWP).  The 
amount  budgeted  for  the  work  actually  performed 
(6:6)  . 

4.  Budgeted  Cost  for  work  Schedulea  ^BCWS).  The 
amount  budgeted  for  the  work  scheduled  to  be 
completed  during  a  given  time  period  (6:6). 

5.  Estimate  at  Completion  (EAC).  The  actual  costs 
of  the  work  performed  to  date  plus  an  estimate  of  the 
costs  to  complete  the  remaining  authorized  work 
(6:6)  . 

(These  terms  were  defined  in  the  first  chapter.)  The 
schedule  variance  can  be  determined  by  comparing  BCWS  and 
BCWP  and  indicates  whether  work  Is  behind  or  ahead  of 
schedule.  Although  the  schedule  variance  will  be  stated  in 
terms  of  dollars  rather  than  in  units  of  time,  it  is  an 
early  "flag"  of  potential  problems  and  Indicates  the  need 
for  more  detailed  analysis.  Similarly,  the  cost  variance 
can  be  determined  by  comparing  BCWP  and  ACWP  (2:34;  15:31). 


'“he  system  employed  by  the  contractor  must  provide  a 
summary  of  these  elements  and  the  associated  variances,  and 
an  analysis  of  variances  if  predetermined  thresholds  are 
exceeded  (9;17). 

Revisions  and  Access  to  Data .  These  criteria 
"require  the  contractor  to  do  basically  two  things:  (1) 
maintain  a  valid  performance  measurement  baseline,  and  (2) 
provide  government  access  to  Internal  data"  (9:17). 

Reporting .  Interestingly,  the  C/SCSC  has  no 
requirements  for  external  reporting.  "Summary  data  from 
the  [contractor's]  internal  system  are  reported  to  the 
government  through  the  Cost  Performance  Report  (CPR),  as 
specified  on  the  Contract  Data  Requirements  List"  (1:46). 
Typically,  the  CPR  Is  submitted  monthly  to  the  government 
program  office  and  reports  summary  information  collected  by 
the  contractor's  C/SCSC-compl iant  internal  management 


system.  Normally,  detailed  information  is  not  necessary 
(unless  a  problem  exists)  because  C/SCSC  has  "guaranteed" 
that  objective  cost  and  schedule  information  has  been 
provided  by  the  contractor.  This  permits  program 
management  to  focus  their  attention  on  problem  areas  and 
take  timely  corrective  action.  In  addition  to  confirming 
problems,  information  contained  in  the  CPR  is  used  to 
project  variance  trends  to  contract  completion  and  to 
forecast  completion  dates  and  final  contract  costs  (1:46) 
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As  mentioned  previously,  summary  data  from  a 
contractor's  internal  management  control  system  are 
reported  to  the  government  through  the  CPR.  One  of  the  key 
data  items  required  by  this  report  is  the  contractor's 
Latest  Revised  Estimate  of  the  contract's  cost  at 
completion,  or  EAC.  These  estimates  are  useful  tools  for 
the  program  office  to  verify  funding  requirements 
identified  by  a  contractor  on  the  Contract  Funds  Status 
Report  (1:46-47).  Additionally,  "EAC's  are  a  primary  input 
into  the  Selected  Acquisition  Reports  (SAR's)  which  go  to 
major  acquisition  commands  and  eventually  end  up  in 
Congress "  ( 5  :  i i  )  . 

The  contractor's  EAC  should  consist  of  the  actual  cost 
of  work  performed  to  date  plus  the  latest  estimate  of  the 
cost  to  complete  the  remaining  work.  The  joint 
Implementat ion  guide  provides  an  explicit  checklist  for 
reviewing  the  procedures  and  data  used  by  contractors  to 
generate  EAC's.  Numerous  methods  exist  for  generating 
EAC's  and  the  criteria  do  not  specify  a  particular  method 
that  the  contractor  must  use.  A  number  of  methods  for 
generating  EACs  are  discussed  by  Bowman  including; 
indices,  weighting  systems,  expectations  theory,  and  a 
variety  of  hybrid  techniques  (4).  However,  whatever 
methodology  is  selected  by  the  contractor,  it  must  be 
rational,  consistent,  and  reconcilable  (6:47). 


The  important  words  in  the  last  sentence  are 
''rational/'  "consistent,"  and  "reconcilable."  There 
appears  to  be  no  one  "best"  way  to  compute  EACs--each 
technique  will  normally  yield  a  different  estimate  at 
completion.  The  problem  is  that  numbers  can  be  manipulated 
mathematically  to  produce  the  desired  results,  accurate  or 
not.  Therefore,  it  is  important  that,  whatever  method  is 
selected,  it  must  be  rational;  any  technique  that  produces 
an  EAC  which  approaches  the  true  value  should  be  favored 
(5:2,23).  As  well  as  being  rational,  the  method  must  be 
applied  consistently.  "If  the  methods  change  (regardless 
of  the  reason)  then  a  rubber  baseline  is  generated .. .and 
the  resultant  EAC  loses  its  meaning"  (5:23).  Finally,  the 
contractor's  estimates  of  costs  at  completion  must  be 
reconcilable  with  cost  data  reported  to  the  government 
program  office  (6:47). 

In  the  search  for  the  "best"  method  of  generating  an 
EAC,  Holeman  points  out  "Emphasis  needs  to  be  placed  in 
this  area  to  first  determine  whether  a  reasonable 
quantitative  approach  Is  available  and  then  decide  the 
limitations  to  this  approach"  (11:41).  However,  there 
seems  to  be  very  little  research  in  this  area  to  date. 

Price  examined  six  automated  techniques  for  computing  EACs 
(four  of  which  are  the  basis  of  this  research)  used  by 
AFWAL's  Cost  Performance  Report  Analysis  program.  The 
results  of  his  research  indicate  the  EAC  method  using  a 


weighted  Cost/Performance  index  and  Schedule/Perf ormance 
Index  was  the  "best  predictor"  (explained  the  highest 
percentage  of  variation),  while  the  method  based  on  the 
monthly  Cost/Performance  Index  was  the  poorest  predictor 
(15:31-32).  (These  techniques  correspond  to  EAC4  and  EACl, 
respectively,  of  this  report.)  Aside  from  Price's  research 
(which  was  limited  to  cost  data  for  on-going  research  and 
development  programs),  the  literature  review  revealed 
little  additional  information  regarding  the  accuracy  or 
applicability  of  various  EAC  methods. 

Monte  Carlo  Simulation 

"Monte  Carlo  methods  comprise  that  branch  of 
experimental  mathematics  which  is  concerned  with 
experiments  on  random  numbers"  (10:2).  The  use  of  Monte 
Carlo  analysis  as  a  research  tool  has  its  origins  in  the 
1940 's  when  John  von  Neumann  and  S.  Ulam  simulated  the 
probabilistic  problems  associated  with  the  shielding  of 
nuclear  devices;  the  problems  were  too  risky  and  expensive 
to  solve  experimentally,  and  too  complicated  to  solve 
analytically  (10:8;  16:135). 

There  are  two  types  of  problems  handled  by  Monte  Carlo 
methods--probabilistic  and  deterministic.  In  either  case, 
the  objective  of  most  Monte  Carlo  analysis  is  to  estimate 
the  unknown  value  of  some  parameter  associated  with  some 
population  distribution  (10:2,16).  Hammersley  describes 
the  case  of  a  probabilistic  problem  (which  is  of  primary 


concern  to  this  research): 


the  simplest  Monte  Carlo  approach  Is  to  observe 
random  numbers,  chosen  in  such  a  way  that  they 
directly  simulate  the  physical  random  processes  of 
the  original  problem,  and  to  infer  the  desired 
solution  from  the  behaviour  of  these  random  numbers 
(10:2)  . 

Thus,  the  common  idea  behind  the  collection  of  techniques 

referred  to  as  "Monte  Carlo  analysis"  is  "approximating  the 

solution  to  a  problem  by  sampling  from  a  random  process" 

(17:135-136).  Stated  differently  by  Hammersley: 

the  essential  feature  common  to  all  Monte  Carlo 
computations  is  that  at  some  point  we  have  to 
substitute  for  a  random  variable  a  corresponding  set 
of  actual  values  having  the  statistical  properties  of 
the  random  variable  (10:25). 

Summary 

This  chapter  presented  bacicground  Information  on  three 
areas  relevant  to  this  research  project.  First,  the 
Cost/Schedule  Control  Systems  Criteria,  or  C/SCSC,  was 
examined.  The  C/SCSC  describe  the  various  criteria,  or 
standards,  that  a  contractor's  internal  management  control 
system  must  meet  to  be  acceptable  to  the  government.  An 
Important  aspect  of  the  C/SCSC  is  that  it  is  not  a 
system--it  does  not  impose  a  particular  management  control 
system  on  the  contractor . 

Second,  literature  pertaining  to  estimates  at 
completion  of  contract  costs  was  presented.  Estimates  at 
completion,  EACs,  should  Include  the  contractor's  actual 
cost  of  wor)c  completed  to  date  plus  an  estimate  of  the  cost 
to  complete  the  remaining  contractual  work.  There  appears 


-20- 


to  be  very  little  research  in  the  area  of  identifying  a 


"best"  method  for  calculating  EACs . 

Finally,  Monte  Carlo  simulation  was  addressed.  The 
types  of  problems  handled  by  Monte  Carlo  methods  include 
probabilistic  and  deterministic.  In  both  cases,  the  usual 
objective  is  to  approximate  the  unknown  value  of  a 
population  parameter  by  sampling  from  a  random  process. 
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III .  METHODOLOGY 


Introduction 


To  analyze  the  robustness  and  compute  the  statistical 


moments  associated  with  each  o£  the  estimate  at  completion 


techniques,  it  would  have  been  ideal  to  have  had  actual 


data  from  completed  programs.  However,  there  was 


Insufficient  time  to  collect  actual  data  and  manipulate  It 


into  a  usable  format.  Therefore,  Monte  Carlo  simulation 


methods  were  used  to  generate  sufficient  data  to  answer  the 


research  questions. 


This  research  investigated  samples  from  18  simulated 


populations  corresponding  to  the  possible  combinations  of 


the  following;  a  low  and  high  range  for  the  monthly  BCWP 


values,  means  of  -10,  -20,  and  -30  'percent)  for  the  cost 


variance  percentage  distributions  (cost  overrun  scenarios). 


and  means  of  0,  -5,  and  -10  for  the  schedule  variance 


percentage  distributions  (on  schedule  and  two  behind 


schedule  scenarios).  The  computer  program  used  for  most  of 


this  research  was  S.  S  is  a  UNIX-based  software  system 


that  provides  the  user  with  "an  interactive  environment  fo^ 


data  analysis  and  graphics"  (3;i).  S  was  considered  to  be 


particularly  suitable  for  this  application  because  of  its 


ease  of  use  and  its  ability  to  generate  random  samples  from 


a  variety  of  distributions. 
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since  Monte  Carlo  simulation  methods  were  used  to 
generate  the  required  data  values  for  this  research,  it  was 
necessary  to  make  certain  simplifying  assumptions 
concerning  how  much  data  was  to  be  generated  and  how  it  was 
to  be  accomplished.  The  first  assumption  was  that  all 
simulated  "programs ,  **  or  contracts,  span  60  months.  Hence, 
data  values  were  generated  as  if  each  program  lasted  five 
year3--it  began  in  month  one  and  ended  in  month  sixty. 

The  second  assumption  was  that  the  monthly  BCWP  values 
lie  in  one  of  two  ranges,  or  3trata--a  "low"  range 
consisting  of  monthly  values  from  $0  to  $500,000  and  a 
"high"  range  consisting  of  values  from  $500,000  to 
$15,000,000.  This  stratification  makes  It  possible  to 
evaluate  the  capabilities  of  each  of  the  BAC  techniques  for 
low-  and  high-value  contracts. 

The  third  assumption  concerns  the  distributions  of  the 
cost  and  schedule  variance  percentages.  It  was  assumed 
that  both  were  normally  distributed  with  a  standard 
deviation  of  2.5  percent.  Examination  of  the  database  used 
by  Price  appears  to  support  this  assumption--whi le  smaller 
valued,  shorter  duration  contracts  exhibited  larger 
standard  deviations  In  these  parameters;  the  standard 
deviations  were  significantly  reduced  for  larger  valued 
contracts  for  which  more  data  was  available. 


Th«i  tourth  assumption  was  that  the  starting  month  tor 
calculations  (the  simulated  last  month  for  which  data  was 
available)  was  uniformly  distributed  between  months  4  and 
59.  This  assumption  was  made  because  Eq  (7)  (EAC2)  uses 
the  three  month  Cos t/Per f ormance  Index,  THREE  MON  CPI, 
which  requires  at  least  3  months  of  data.  Also,  it  makes 
little  sense  to  calculate  an  EAC  if  the  program  has  been 
completed  (month  60). 

Finally,  it  was  necessary  to  simulate  values  for  the 
actual  and  budgeted  costs  at  completion.  Therefore,  it  was 
assumed  that  the  last  available  ACWP  value,  ACWP(60), 
represented  the  actual  contract  cost  at  completion. 
Similarly,  the  last  available  BCWP  value,  BCWP(60),  was 
assumed  to  simulate  the  contract's  budgeted  cost  at 
completion,  or  BAC .  This  method  is  similar  to  that  used  by 
Price  ( 15 ) . 

Robustness  Measure 

As  discussed  In  Chapter  I,  the  definition  of  a 
"robust"  estimate  is  subjective;  It  depends  upon  who  Is 
using  the  estimate  and  their  criteria  for  what  they 
consider  an  acceptable  result  (13:1).  This  research  is 
interested  In  comparing  the  robustness  of  various 
techniques  for  estimating  a  contract's  cost  at  completion; 
therefore,  a  measure  of  robustness  Is  needed  before  any 
comparisons  can  be  made.  For  the  purposes  of  this 
research,  the  following  operational  definition  of  a 


tobu3tn«33  mea3ure,  RM,  will  be  iJ3ed: 


RMn  =  (EACn  -  ACWP ( 60 ) ) /ACWP ( 60 )  (12) 

where 

n  corre3pond3  to  one  of  the  four  EAC  technlque3 

and 

ACWP(60)  represents  the  contract  completion  cost 

Therefore,  a  value  of  -.15  for  RMl  Indicates  that  the  EACl 
technique  resulted  In  an  estimate  at  completion  that  was  15 
percent  below  the  simulated  actual  contract  completion 
cost.  Similarly,  a  value  of  .10  for  the  robustness  measure 
indicates  an  estimate  that  was  10  percent  above  the  actual 
value . 

Methodology  and  §.  Program 

After  making  the  above  assumptions  and  defining  the 
robustness  measure,  it  was  necessary  to  develop  an  S 
program  that  would  simulate  actual  data  found  in  a  Cost 
Performance  Report  and  perform  the  EAC  and  robustness 
calculations  for  each  of  the  18  populations.  The  program 
developed  for  this  purpose  can  oe  found  in  Appendix  A  and 
each  of  the  steps  in  the  process  Is  discussed  below. 

Since  each  simulated  program  was  assumed  to  last  60 
months,  the  first  step  was  to  generate  60  monthly  BCWP 
values  for  the  selected  range  (low  or  high)  from  a  uniform 
distribution.  This  was  accomplished  by  using  the  random 
number  generator  located  in  S  which  permits  random  sampling 


from  a  uniform  distribution  with  specified  lower  and  upper 
bounds.  These  values  were  then  summed  to  obtain  the 
cumulative  BCWP  values  needed  in  later  calculations. 

The  second  step  was  to  generate  60  monthly  values  from 
normal  distributions  for  both  the  cost  variance  percentage 
and  schedule  variance  percentage.  Again,  a  random  number 
generator  contained  in  S  was  used  to  randomly  sample  60 
values  for  each  variable  from  separate  normal  distributions 
with  standard  deviations  of  2.5  percent  and  means 
corresponding  to  the  selected  cost  and  schedule  overrun 
scenar ios . 

To  compute  the  various  indices  and  EACs,  it  was 
necessary  to  have  the  cumulative  monthly  values  for  ACWP 
and  BCWS .  Therefore,  the  third  step  was  to  calculate 
values  for  these  variables  using  the  cumulative  BCWP 
values,  the  monthly  cost  variance  percentages,  and  the 
monthly  schedule  variance  percentages,  generated  in  the 
preceding  steps.  This  was  done  by  rearranging  Eqs  (1)  and 
(5)  to  obtain  the  following: 

ACWP(m)  =  BCWP(m)  *  (1.  -  COST  VAR  %(m)/100.)  (13) 

and 

BCWS(m)  =  BCWP(m)/(l.  +  SCH  VAR  %(m)/100.)  (14) 

where 

m  =  a  particular  month  from  1  to  60 
For  example,  the  cumulative  monthly  value  of  BCWP  for  month 


25,  BCWP(25),  was  used  In  addition  to  the  randomly 
generated  values  for  the  cost  and  schedule  variance 
percentages  for  that  month  to  compute  ACWP  and  BCWS  for 
month  25,  ACWP(25)  and  BCWS(25)  respectively. 

With  the  60  cumulative  monthly  values  for  BCWP,  ACWP, 
and  BCWS,  the  fifth  step  was  to  determine  the  starting 
month  for  calculat ions--the  simulated  last  month  for  which 
data  was  available.  This  was  done  so  that  EACs  would  be 
computed  for  programs  simulated  in  various  stages  of 
completion.  A  sequence  of  integers  from  4  to  59  was 
generated  and  a  single  value  was  randomly  chosen  to 
represent  the  last  month  for  which  data  was  available;  this 
month  was  designated  by  the  Indicator,  "(CUM),"  in  Eqs 
(1)-(12) . 

The  sixth  step  was  to  use  the  data  generated  by  the 
previous  steps  to  calculate  the  various  indices  given  by 
Eqs  (2)-(4).  Then,  these  indices  were  used  in  addition  to 
other  data  values  to  compute  estimates  of  the  contract's 
cost  at  completion  for  each  of  the  four  EAC  models,  Eqs 
(6)-( 10) . 

After  calculating  the  EACs  and  having  the  simulated 
actual  contract  cost  at  completion,  ACWP(60),  it  was 
possible  to  compute  the  robustness  measures  associated  with 
each  of  the  EAC  techniques.  Thus,  the  final  step  was  to 
use  Eq  (12)  to  determine  the  robustness  measure  for  each  of 
the  four  estimate  at  completion  techniques. 


The  seven  steps  described  above  were  replicated  300 
times  and  the  pertinent  data  stored  In  a  matrix.  Hence, 
estimates  at  completion  and  robustness  measures  for  each  of 
the  EAC  models  were  computed  for  300  programs,  or 
contracts.  In  various  stages  of  completion.  A  sample  size 
of  300  should  be  sufficiently  large  so  that  summary 
statistics  associated  with  the  samples  can  be  assumed  to 
have  been  derived  from  approximately  normal  distributions 
{7:199;  13:5-6). 

In  order  to  address  the  research  questions,  it  was 
necessary  to  compute  the  statistical  moments  of  the 
distributions  associated  with  each  of  the  four  EAC 
techniques.  Therefore,  the  mean,  standard  deviation,  and 
variance,  were  calculated  for  each  of  the  EAC  and 
robustness  measure  distributions.  In  the  case  of  the 
variance  (and  hence,  the  standard  deviation),  S  assumes  it 
Is  wor)<lng  with  a  population  rather  than  a  sample.  As  a 
result,  when  computing  the  sample  variances,  it  was 
necessary  to  multiply  the  variances  computed  in  S  by  the 
expression,  "n/(n-l),”  where  "n"  is  the  sample  slze--in 
this  particular  case,  n=300. 

The  procedures  discussed  In  this  section  were 
performed  for  samples  from  each  of  the  18  populations. 


Introduction 


IV.  Results  and  Discussion 


This  chapter  presents  the  results  of  the  analysis 
performed  on  the  data  generated  by  the  procedures  described 
in  the  previous  chapter.  The  results  are  discussed  in  the 
order  of  the  research  questions  outlined  in  Chapter  I. 
First,  the  statistical  moments  associated  with  the 
distributions  generated  by  each  of  the  four  EAC  techniques 
under  various  simulated  cost  and  schedule  overrun  scenarios 
is  presented.  Next,  differences  in  the  statistical  moments 
are  discussed--both  the  differences  between  the  four 
techniques  for  a  given  population,  and  for  a  particular 
technique  between  various  populations.  Finally,  the 
robustness  of  the  four  EAC  models  is  discussed. 

Due  to  the  large  number  of  populations  simulated  in 
this  research,  the  following  nomenclature  Is  used  to 
identify  the  individual  populations.  Each  population  is 
distinguished  with  a  four  or  five  character  label.  The 
first  two  digits  Identify  the  cost  overrun  scenario,  the 
third  character  represents  the  BCWP  range,  and  the  fourth 
(and  fifth,  if  required)  digit  identifies  the  schedule 
overrun  scenario.  For  example,  "20L10"  represents  the 
simulated  population  with  a  cost  variance  percentage  of  -20 
percent,  BCWP  values  from  the  "low"  range,  and  a  schedule 
variance  percentage  of  -10  percent.  Similarly,  "lOHO" 
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represents  the  simulated  population  with  a  cost  variance 


percentage  of  -10  percent,  BCWP  values  from  the  "high" 
range,  and  a  schedule  variance  percentage  of  0  percent  (on 
schedule ) . 

Results  of  Research  Question  1. 

The  first  research  question  was  to  determine  the 
statistical  moments  of  the  EAC  distributions  that  resulted 
from  generating  estimates  at  completion  with  each  of  the 
four  techniques  for  each  of  the  simulated  populations.  The 
statistical  moments  associated  with  the  four  EAC 
techniques,  EAC1-EAC4,  can  be  found  in  Tables  1-4, 
respectively,  of  Appendix  B.  Additionally,  the  summary 
statistics  for  each  of  the  individual  populations  are 
located  in  Tables  9-26  of  Appendix  D. 

Results  of  Research  Question  2. 

The  second  research  question  asked  If  there  are 
significant  differences  in  the  means  and  variances  of  the 
distributions  generated  by  different  EAC  models  for  a  given 
cost  and  schedule  overrun  scenario  and,  also,  by  the  same 
model  for  various  scenarios.  Examination  of  Tables  1-4  of 
Appendix  B  reveals  that  the  means  of  the  distributions 
associated  with  EAC2,  EAC3,  and  EAC4,  are  comparable--not 
only  for  a  particular  population,  but  across  the  various 
populations  as  well.  However,  for  a  particular  population 
and  over  all  the  populations,  the  means  associated  with  the 


EACl  distributions  were  consistently  larger  than  those  of 
the  other  three  techniques.  This  indicates  that,  in 
general,  use  of  the  EACl  model  results  in  larger  estimates 
at  completion  in  a  cost  overrun  situation  than  the  other 
techniques  researched.  Each  of  the  techniques  responded  to 
the  increased  cost  overruns  by  increased  estimates  at 
completion.  In  other  words,  as  the  cost  variance 
percentage  "increased"  from  -10  percent  to  -30  percent, 
each  of  the  techniques  "recognized"  a  larger  cost  overrun 
and  increased  its  estimate  at  completion.  However,  the 
different  values  for  the  schedule  variance  percentage  had 
no  noticeable  impact  on  the  estimates  of  the  EAC4 
model--the  only  EAC  technique  examined  that  incorporates 
the  schedule  variance  percentage.  This  may  be  attributable 
to  the  smaller  values  selected  for  the  schedule  variance 
percentage . 

In  the  case  of  the  variances.  Tables  1-4  indicate  that 
the  variances  of  the  EAC  distributions  associated  with  EAC3 
are  consistently  lower  than  those  of  the  other  techniques, 
though  the  variances  of  the  EAC4  distributions  are  very 
similar.  This  is  true  for  any  particular  population  and 
overall.  The  variances  for  the  EAC2  distributions  are 
slightly  larger  than  those  of  EAC3  and  EAC4,  but  are  still 
much  smaller  than  those  of  EACl;  the  variances  of  the  EACl 
distributions  are  consistently  much  larger  than  for  the 
other  models.  This  behavior  most  likely  results  from  the 
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performance  index  which  may  change  dramatically  from  one 
month  to  the  next;  the  other  techniques  all  use  some  form 
of  "smoothing"  to  lessen  the  Impact  of  month-to-month 
changes.  As  might  be  expected,  when  the  cost  overruns 
increased,  the  variances  associated  with  each  of  the  EAC 
models  increased  also.  Again,  the  variances  of  the  EAC4 
distributions  seemed  unaffected  by  the  increased  schedule 
variance  percentages. 

Results  of  Research  Question 

The  third  research  question  was  concerned  with 
identifying  the  EAC  technique  which  is  most  robust  for  a 
particular  cost  and  schedule  overrun  scenario.  The  summary 
statistics  for  the  robustness  measures  can  be  found  in 
Tables  5-8  of  Appendix  C.  One  characteristic  of  all  the 
EAC  models  is  immediately  apparent  from  these  tables--for 
every  simulated  population,  all  of  the  models 
underestimated  the  simulated  completion  cost.  This  is 
indicated  by  negative  values  for  all  of  the  means 
associated  with  the  robustness  measure  distributions. 

That  is  not  to  say  there  were  no  instances  in  which  the 
models  overestimated  the  completion  cost,  but,  in  general, 
the  tendency  for  each  of  the  EAC  techniques  was  to 
underestimate  the  final  cost. 

Further  examination  of  Tables  5-8  yields  the  following 
results  concerning  the  means  and  variances  of  the 


robustness  measure  distributions.  The  means  for  the 
distributions  ranged  in  value  from  approximately  -.14  to 
-.43  which  indicates  estimates  at  completion  of  contract 
costs  from  14  to  43  percent  below  the  "actual"  cost.  The 
means  of  the  robustness  measure  distributions  associated 
with  EACl  were  consistently  the  lowest;  this  was  expected 
since  the  means  of  the  EAC  distributions  for  EACl  were  the 
largest.  The  means  of  the  robustness  measure  distributions 
resulting  from  EAC3  and  EAC4  were  comparable  though 
slightly  larger  than  those  of  EAC2 .  This  behavior  is 
evident  for  any  of  the  particular  populations,  as  well  as 
across  all  the  populations. 

There  are  several  interesting  observations  regarding 
the  behavior  of  the  robustness  measure  means  as  the  cost 
variance  percentage  increased  (increased  overruns).  When 
the  cost  overruns  increased,  the  means  of  the  robustness 
measure  distributions  for  each  of  the  EAC  techniques 
decreased  (smaller  absolute  value).  As  mentioned  above, 
the  means  for  EAC2,  EAC3,  and  EAC4,  are  similar  and, 
interestingly,  they  all  followed  the  same  pattern  of 
reduction  (approximately  -.42  to  -.31)  when  the  cost 
variance  percentage  increased  for  both  the  low  and  high 
range  of  the  BCWP  values.  This  suggests  that  each  of  these 
techniques,  EAC2-EAC4,  is  more  sensitive  to  the  magnitude 
of  the  cost  variance  percentage  than  the  BCWP  value. 


However,  the  same  is  not  true  for  EACl. 


As  the  cost 


variance  percentage  Increased,  the  robustness  measure  means 
of  EACl  decreased  much  less  for  the  high  BCWP  range  than 
for  the  low  suggesting  that  EACl  is  affected  not  only  by 


changes  in  the  cost  variance  percentage,  but  changes  in  the 
magnitude  of  the  BCWP  values  as  well. 

The  variances  displayed  similar  behavior  patterns.  In 
all  cases,  as  the  cost  overruns  increased,  the  variances  of 
the  robustness  measure  distributions  associated  with  each 
of  the  EAC  techniques  increased.  The  variances  for  EAC3 
and  EAC4  were  comparable  and,  generally,  the  lowest  for  all 
of  the  populations;  though  the  variances  for  EAC2  were  only 
slightly  higher.  Again,  as  the  cost  variance  percentage 
increased,  the  variances  for  EAC2-EAC4  followed  a  similar 
pattern  of  Increase  for  both  ranges  of  the  BCWP  values. 
Additionally,  this  pattern  was  over  a  relatively  narrow 
band  (.09  to  .13)  for  EAC3  and  EAC4.  The  variances 
associated  with  EACl  were  consistently  the  largest  and 
increased  dramatically  as  the  cost  variance  percentage 
increased  for  the  low  range  of  BCWP  values. 

Summary 

For  the  given  set  of  cost  and  schedule  overrun 
scenarios  selected  for  this  research  project,  the  tendency 
of  all  four  of  the  EAC  models  was  to  underestimate  the 
contract's  final  cost.  The  values  for  the  means  of  the 
robustness  measure  distributions  ranged  from  approximately 
-.14  to  -.43  indicating  underestimates  ranging  from  14  to 
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43  percent  o£  the  contract's  cost  at  completion.  For  a 
particular  population,  the  means  associated  with  EACl  were 
lowest  of  the  four  techniques  in  all  cases,  but  the 
corresponding  variances  were  also  significantly  larger.  Of 
the  four  models,  EACl  was  the  only  technique  seemingly 
sensitive  to  the  magnitude  of  the  BCWP  values  in  addition 
to  changes  in  the  cost  variance  percentages.  The  variances 
of  the  robustness  measures  associated  with  EAC2,  EAC3,  and 
EAC4,  were  comparable  to  one  another  and  relatively  stable 
across  all  of  the  simulated  populations. 


V.  Conclusions 


Review  of  the  Hvpothes Is 

The  research  hypothesis  was  that  a  statistical 
evaluation  of  each  of  the  BAC  models  for  generating 
estimates  at  completion  of  contract  costs  may  identify  a 
robust  technique.  This  hypothesis  was  tested  using  Monte 
Carlo  simulation  techniques  to  calculate  estimates  at 
completion  for  a  number  of  simulated  cost  and  schedule 
overrun  scenarios. 

Conclus ions 

The  results  presented  in  the  preceding  chapter  suggest 
that  none  of  the  selected  EAC  techniques  are  robust  for  the 
cost  and  schedule  overrun  scenarios  that  were  examined. 
Though  a  measure's  "robustness"  Is  subjective,  for 
practical  purposes,  it  would  be  desirable  to  have  an 
estimate  at  completion  technique  that  could  reliably 
estimate  a  contract's  final  cost  within  five  or  ten  percent 
over  a  wide  range  of  situations.  All  of  the  EAC  techniques 
in  this  research  failed  to  approach  this  desired  threshold. 
The  statistical  means  of  the  robustness  measure 
distributions  for  EAC2,  EAC3,  and  EAC4,  indicate  they  were 
unable  to  consistently  predict  a  contract's  completion  cost 
within  30  percent.  However,  the  variances  of  the 
robustness  measures  associated  with  these  techniques  were 
relatively  stable  across  all  of  the  simulated  populations 


suggesting  the  possibility  that  they  could  be  modified  to 
more  accurately  predict  the  final  contract  cost;  thus, 
making  them  more  robust.  The  means  of  the  robustness 
measure  distributions  for  EACl  were  lower  than  those  of  the 
other  three  models  for  Individual  populations;  however,  the 
large  variances  associated  with  EACl  and  it's  apparent 
sensitivity  to  the  magnitude  of  the  BCWP  values  make  it  a 
very  unreliable  predictor  of  a  contract's  completion  cost. 

Additionally,  for  all  of  the  simulated  cost  and 
schedule  overrun  scenarios,  the  tendency  of  each  of  the  EAC 
techniques  was  to  underestimate  the  contract's  final  cost. 
This  tendency  may  have  serious  implications  for  the  program 
office  in  this  time  of  increased  media  attention. 
Congressional  oversight,  and  growing  budget  deficits. 

Recommendations  for  Future  Research 

These  conclusions  were  based  on  Monte  Carlo  simulation 
methods  and,  as  with  any  simulation,  it  may  be  appropriate 
to  question  their  appl icabi 1 ity  to  the  "real  world." 
Therefore,  future  research  could  analyze  these  EAC 
techniques  with  actual  data  from  completed  programs  to 
verify  the  reasonableness  of  these  conclusions  and,  hence, 
the  methods  that  were  used  in  this  research.  Also,  it  may 
be  beneficial  to  apply  this  methodology  to  other  EAC 
techniques,  or  modifications  to  the  models  used  in  this 
research,  to  determine  if  more  robust  models  exist. 

Lastly,  future  research  could  analyze  these  and  other 
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techniques  In  terms  ot  acquisition  phase  (tull  scale 
development,  production,  etc.),  contract  type,  percentage 
of  contract  completed,  or  type  of  contract  deliverable. 
Research  In  these  areas  could  Identify  robust  methods  for 
calculating  estlnvates  at  completion  of  contract  costs,  or 
determine  which  EAC  techniques  are  best  suited  for  a 
particular  set  of  conditions. 
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Append  lx  A:  S  Program 


Author  : 

Capt 

Darren  E.  Morgan 

Written : 

May 

1987 

» 

opt ions(echo=2) 


POPULATION  FOR  THIS  RUN:  20H10 


»  SET  LOWER  AND  UPPER  LIMITS  FOR 

I  MONTHLY  BCWP  VALUES 

i 

#  LOWER  LIMIT 
lowl im_500000 . 

i 

»  UPPER  LIMIT 
upl im_15000000  . 


I  SET  VALUES  FOR  MEAN  AND  STD.  DEV. 

»  OF  COST  VARIANCE  PERCENTAGES 

»  (NEG.  VALUE  =>  COST  OVERRUN) 

» 

*  MEAN 

cvmu_- 20 . 

• 

#  STD.  DEV. 

CVS  igma_2  .  5 

» 

• 

i  SET  VALUES  FOR  MEAN  AND  STD.  DEV. 

»  OF  SCHEDULE  VARIANCE  PERCENTAGES 

i  (NEG,  VALUE  =>  BEHIND  SCHEDULE) 

• 

«  MEAN 

3vrou_-10 . 

• 

«  STD.  DEV. 

CVS  igma_2 . 5 


SET-UP  MATRIX  TO  HOLD  DATA  VALUES 


-39 


n_300 

ca3el_matr lx ( 0, n, 15, byrow=TRUE ) 

» 

»  CREATE  LOOP  TO  COMPUTE  n=300  VALUES 
for  (row  in  l:n)  [ 

» 

»  RANDOMLY  SELECT  60  MONTHLY  VALUES  FOR  BCWP 
»  FROM  A  UNIFORM  DISTRIBUTION  WITH  THE 

#  LOWER  AND  UPPER  BOUNDS  SET  ABOVE 

» 

bcwplnlt_runl f ( 60 , lowl im, upllm) 

» 

»  CALCULATE  CUMULATIVE  VALUES  FOR  BCWP 

bcwp_cumsum( bcwpinit ) 

» 

« 

#  RANDOMLY  SELECT  60  MONTHLY  VALUES  FOR 

»  COST  VARIANCE  PERCENTAGE  AND  SCHEDULE 

#  VARIANCE  PERCENTAGE  FROM  NORMAL 

»  DISTRIBUTIONS  WITH  THE  MEANS  AND  STANDARD 

»  DEVIATIONS  SET  ABOVE 

» 

cvpctq_rnorm( 60,cvmu, cvs Igma ) 
svpctg_rnorm(  60,  svmu  svsignva) 

I 

» 

»  CALCULATE  THE  CUMULATIVE  MONTHLY  VALUES 
»  FOR  ACWP 

» 

acwp_bcwp* { 1 . -cvpctg/100  .  ) 

» 

ft  CALCULATE  THE  CUMULATIVE  MONTHLY  VALUES 
It  FOR  BCWS 

# 

bcws_bcwp/ ( 1 . +3vpctg/100 . ) 

» 

» 

It  SET  UP  A  SEQUENCE  OF  VALUES  FROM  4-59 
It  REPRESENTING  POSSIBLE  STARTING  MONTHS 

»  FOR  ESTIMATE  AT  COMPLETION  CALCULATIONS 

» 

mths_3eq( 4,59,1) 

» 

* 

#  RANDOMLY  SELECT  A  STARTING  MONTH  FROM  THE 

•  ABOVE  SEQUENCE  TO  SERVE  AS  THE  STARTING 


»  MONTH  FOR  ESTIMATE  AT  COMPLETION  CALCULATIONS 

# 

3tpt_3ample ( mths , 1 ) 

» 

» 

#  CALCULATE  THE  CURRENT  MONTH,  THREE  MONTH 

»  AND  CUMULATIVE  COST  PERFORMANCE  INDICES 

# 

cmcpl_( bcwp[ stpt ] -bcwpl 3tpt-l 1 )/ (acwpt  stpt ] - 
acwptstpt-l 1 ) 

thtncpi_(  bcwptstpt  ]  -bcwp(  stpt-3  } )  /  (acwp(  stpt  ]  - 
acwpt3tpt-3 1 ) 

cumcpi_bcwp[ stpt 1 /acwpt  stpt 1 

#  COMPUTE  ESTIMATES  AT  COMPLETION  USING 

»  THE  FOUR  EAC  TECHNIQUES 

» 

eacl_acwp [stpt ]  +  ( acwp ( stpt ] -bcwpt  stpt ] ) /cmcpi 
eac2_acwp[ stpt I f ( acwp [ stpt 1 -bcwpt  stpt ] ) /thmcpi 
eac3_acwp[ stpt 1  +  (acwp  t  stpt 1 -bcwpt  stpt ] ) /cumcpi 
etc_( 100 . -cvpctgt  stpt 1  - . 75*svpctg( stpt 1 ) * 

( scwpt stpt ] -bcwp [ stpt ] )/100. 
eac4_acwp(  stpt  1  +-etc 
» 

» 

«  COMPUTE  ROBUSTNESS  MEASURES  FOR  EACH 
»  OF  THE  EAC  TECHNIQUES 

I 

rml_( eacl-acwpt  60 1 ) /acwp  1 60 1 
cm2_( eac2-acwp( 601) /acwpt  60 1 
rm3_( eac3-acwp[ 60] )/acwpt60J 
rm4_( eac4-acwp( 601) /acwpt  60 1 

» 

I 

»  SAVE  DATA  IN  MATRIX 

» 

easel t  rowv, 1 ]_uplim 
caseltrowv, 2  I _cvmu 
easel [ rowv, 3 ] _cvs igma 
caseltrowv, 41 _svmu 
easel t  rowv, 5  I _s vs igma 
easel [ rowv, 6 1 _eacl 
easel t rowv, 7 l_eac2 
easel t  rowv, 8 1 _eac3 
caseltrowv, 9 ]_eac4 
easel [ rowv, 10 )_rml 
easel t  rowv, 11 l_rm2 
caseltrowv,  121_rin3 
easel [ rowv, 13 ]_rm4 
easel t  rowv, 14 l_stpt 
easel t  rowv, 15 l_acwp[ 60 1 


» 

» 


CLOSE  ''FOR''  LOOP 


» 

»  COMPUTE  SAMPLE  MEAN,  VARIANCE,  AND  STD.  DEV. 

»  FOR  OUTPUT  ASSOCIATED  WITH  EACH  EAC  TECHNIQUEE 

» 

»  EACl 

mean { case 1[ , 6  ]  ) 

var l_var ( easel [ , 6 1 ) *n/ ( n-1 ) 

varl 

sqrt ( varl ) 

» 

»  EAC  2 

mean ( easel ( ,71) 

var 2_var ( easel [,7] )*n/(n-l) 

var  2 

sqrt ( var 2 ) 

» 

»  EAC  3 

mean ( easel [ , 8  ]  ) 

var 3_var ( easel [,8J )*n/(n-l) 

var  3 

sqrt (var 3 ) 

» 

»  EAC  4 


mean ( case 1( ,91) 

var 4_var ( easel (,91 )*n/(n-l) 

var  4 

sqrt (var 4 ) 

# 

»  COMPUTE  THE  MEAN,  VARIANCE,  AND  STD.  DEV. 
I»  FOR  EACH  OF  THE  ROBUSTNESS  MEASURES 

# 

#  RMl 


mean ( easel ( ,101) 

varrml_var ( easel [ ,101 ) *n/ ( n-1 ) 

varrml 

sqrt ( varrml ) 


» 

I  RM2 

mean ( easel ( ,111) 

varrm2_var ( easel ( ,111 ) *n/( n-1 ) 

varrm2 

sqrt ( var rm2 ) 

« 

I  RM3 

mean ( easel [ ,121) 

varrm3_var (casel(,12] )*n/(n-l) 
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varrm3 

sqrt ( varrm3 ) 

» 

#  RM4 

mean(casel[ ,131) 

varrin4_var  ( easel  [,131  )  *n/  ( n-1 ) 

varrm4 

sqrt ( varrm4 ) 

# 

»  COMPUTE  MEAN,  VARIANCE,  AND  STD.  DEV. 
»  FOR  THE  STARTING  POINT 


* 

mean (easel (,141) 

varstpt_var  (caseK  ,141  )*n/(n-l) 

varstpt 

sqrt ( varstpt ) 

# 

#  COMPUTE  THE  MEAN,  VARIANCE,  AND  STD.  DEV. 

»  FOR  THE  FINAL  ACWP  VALUE  (SIMULATED 

#  FINAL  CONTRACT  COST) 

» 

mean ( easel [,151 ) 

varaewp_var ( easel [ ,151 )*n/(n-l) 

varaewp 

sqrt ( varaewp ) 

# 

» 

#  OUTPUT  DATA  MATRIX 
easel 
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statistical  Momenta  o£  EAC  Dlatr ibutiona 


Table  1 


Statistical  Moments  of  EACl  Distributions 


Dulat ion 

Mean 

Variance 

Std.  Dev. 

lOLO 

10L5 

lOLlO 

10552207  . 
10222382. 
10021643  . 

1.004301e+14 

1.917890e+14 

1. 571839e+14 

10021483 . 
13848790. 
12537300. 

20L0 

20L5 

20L10 

13815946  . 
12113014  . 
12857628. 

1.379802e+15 

6.668586e+14 

2.633370e+14 

37145684. 
25823604 . 
16227658 . 

30L0 

30L5 

30L10 

16645910  . 
16524732. 
16629110  . 

2.687206e+15 

2. 759790e+15 
2.687208e+15 

51838260. 
52533704 . 
51838284 . 

lOHO 

10H5 

lOHlO 

311815616. 
312075456. 
314948480  . 

4.206544e+16 

5.103280e+16 

4.706270e+16 

205098592. 

225904432. 

216939440. 

20H0 

20H5 

20H10 

372786496. 
379096000. 
364959488 . 

8. 399460e+16 
1.036054e+17 

1. 444690e  +  17 

289818240. 
321878000. 
380090656 . 

30H0 

30H5 

30H10 


426566656  . 
416611520  . 
428850560  . 


2. 365435e+17 
1. 871683e+17 
2.361100e+17 


486357000 . 
432629504 . 
485911520 . 


Table  2 


Statistical  Moments 


PoDulat ion 

Mean 

lOLO 

9693292 

10L5 

9484483 

lOLlO 

9529091 

20L0 

11628031 

20L5 

11279937 

20L10 

11533638 

30L0 

13674372 

30L5 

13362036 

30L10 

13646116 

lOHO 

300046656 

10H5 

299169000 

lOHlO 

302756800 

20H0 

356882000 

20H5 

361350304 

20H10 

351440000 

30H0 

414407712 

30H5 

394192512 

30H10 

416608000 

of  EAC2  Distributions 


Var iance 

Std.  Dev. 

2.789606e+13 

2.765948e+13 

2.635388e+13 

5281672. 

5259228. 

5133603. 

3.435702e+13 

4.1288770+13 

4.0927106+13 

5861486. 

6425550. 

6397431. 

4.855310e+13 

5.388900e+13 

4.830540e+13 

6968005. 

7340911. 

6950208. 

2.650512e+16 

2.443547e+16 

2.282570e+16 

162804000  . 
156318464. 
151081776  . 

3.863110e+16 

3. 319137e+16 
3.880280e+16 

196548000. 

182185000. 

196984272. 

5.381430e+16 

4.644060e+16 

5.269840e+16 

231979000. 

215500800. 

229561312. 

Table  3 


Statistical  Moments  of  EAC3  Distributions 


PoDulat ion 

Mean 

Var iance 

Std.  Dev. 

lOLO 

10L5 

lOLlO 

9589969  . 
9351947. 
9404131. 

2. 591912e+13 
2.516910e+13 
2.410610e+13 

5091082. 

5016882. 

4909795. 

20L0 

20L5 

20L10 

11520133. 

11087327. 

11413822. 

3.175133e+13 

3.572390e+13 

3.663916e+13 

5634831. 
5976948. 
6053029 . 

30L0 

30L5 

30L10 

13546857 . 
13200820. 
13516016  . 

4.391314e+13 

4.809944e+13 

4.365570e+13 

6626699. 
6935376 . 
6607248. 

lOHO 

10H5 

lOHlO 

297163000. 
296437280 . 
300169440  . 

2.483380e+16 

2. 314036e+16 
2.168520e+16 

157587408 . 
152119536. 
147259000  . 

20H0 

20H5 

20H10 

353678000. 
357890000. 
345994000  . 

3.510360e+16 

3.065988e+16 

3 . 393490e  +  16 

187359488. 
175099600  . 
184214272. 

30H0 

30H5 

30H10 

407460736. 
389761000  . 
410118400. 

4.561820e+16 

4.108870e+16 

4.486460e-t-16 

213584192. 

202703472. 

211812592. 
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Table  4 


PoDulati 

lOLO 

10L5 

lOLlO 

20L0 

20L5 

20L10 

30L0 

30L5 

30L10 


lOHO 

10H5 

lOHlO 

20H0 

20H5 

20H10 

30H0 

30H5 

30H10 


Statistical  Moments  of  EAC4  Distributions 


Mean 

Var lance 

Std.  Dev. 

9591994. 

9382043. 

9464296. 

2. 594234e+13 
2.534054e+13 
2.443815e+13 

5093362. 
5033939 . 
4943496. 

11519077. 
11146384  . 
11535967. 

3.175832e+13 

3.610728e+13 

3.747280e+13 

5635452. 

6008933. 

6121504. 

13545404. 
13290375. 
13694937  . 

4. 392570e  +  13 
4.878794e+13 

4. 485260e  +  13 

6627646. 
6984836  . 
6697207. 

297225792. 

297388384. 

302044000. 

2.485590e+16 

2.329998e+16 

2.197182e+16 

157657568  . 
152643296, 
148229000. 

353794000. 
359740544. 
349659000  . 

3.515320e+16 

3.099398e+16 

3. 467092e  +  16 

187491840. 

176051000. 

186201280. 

407476000. 

392229000. 

415633000. 

4.561857e+16 

4.167227e+16 

4.609816e+16 

213585000. 

204139000. 

214704800. 

Appendix  c;  statistical  Moments  o£  RM  Distributions 


Table  5 

Statistical  Moments  of  RMl  Distributions 


Population 

Mean 

Var lance 

Std.  Dev. 

lOLO 

-.3596454 

. 377067 

.614058 

10L5 

-.378712 

.6658007 

.815966 

lOLlO 

-.393657 

. 542814 

.  736759 

20L0 

-.2258958 

5.26306 

2.294136 

20L5 

-.3279834 

1.97177 

1.404197 

20L10 

-.284222 

.834229 

.913361 

30L0 

-.1369596 

8.68556 

2.947127 

30L5 

-  .  1432385 

8 . 85594 

2.975894 

30L10 

-.1376086 

8.68554 

2.947124 

lOHO 

-.3901554 

.1616249 

.402026 

10H5 

-.395343 

.206625 

.45456 

lOHlO 

-.3894853 

.1913368 

.4374206 

20H0 

-.3314503 

.271928 

.521467 

20H5 

-.3258837 

.359676 

.59973 

20H10 

-.343044 

. 4570483 

.6760535 

30H0 

-.29367 

.64024 

.80015 

30H5 

- . 297774 

.  565738 

.7521555 

30H10 

-.29059 

.638918 

.799324 

Table  6 


Statistical  Moments  of  RM2  Distributions 


PoDulat ion 

Mean 

Var lance 

Std.  Dev. 

lOLO 

-.4123724 

.1024277 

. 320043 

10L5 

-  .419743 

.  1021418 

.  3195962 

lOLlO 

-.419882 

.096849 

. 3112057 

20L0 

-.36086 

.1002212 

.  3165773 

20L5 

-.367753 

.127554 

. 357147 

20L10 

-  .  3589566 

. 1269422 

.  3562894 

30L0 

-.3063355 

. 1207366 

.  3474717 

30L5 

-.3220447 

.1353002 

.  367832 

30L10 

- . 3075712 

.1201053 

.  346562 

lOHO 

-.413259 

. 1012668 

. 3182245 

10H5 

-.4216004 

.0895257 

.2992085 

lOHlO 

- . 4143805 

. 0826822 

.  287545 

20H0 

-.360192 

. 1244955 

. 352839 

20H5 

-.359221 

.  1012998 

.3182763 

20H10 

-.365342 

.1245045 

. 352852 

30H0 

-.3109253 

.146631 

.3829244 

30H5 

-.336999 

.1256891 

. 3545266 

30H10 

-.3078258 

.  1438957 

.379336 

Table  7 


Statistical  Moments  of  RM3  Distributions 


PoDulat ion 

Mean 

Var lance 

Std.  Dev. 

lOLO 

10L5 

lOLlO 

-.418999 

-.427719 

-.4274625 

.094054 

.0930635 

.0884988 

.  3066823 
.  305063 
.297489 

20L0 

20L5 

20L10 

-.366683 
-.378269 
-  .  3662427 

.0925131 

.1106666 

.111564 

.3041596 
.  3326658 
.334012 

30L0 

30L5 

30L10 

-.3126217 
-  .  3301655 
-.3139995 

.1091058 

.1202938 

.1084363 

.  3303117 
.346834 
. 3292967 

lOHO 

10H5 

lOHlO 

-.4191976 
-.4268747 
-  .  4193546 

.0940038 

.0845853 

.0784237 

. 3066003 
.2908354 
. 2800423 

20H0 

20H5 

20H10 

-.3664585 
-  .  365283 
-.3749374 

.1115042 

.0934018 

.1091143 

.  3339224 
.305617 
. 3303246 

30H0 

30H5 

30H10 

-.3222148 

-.344216 

-.3183693 

.1245656 

.1113792 

.1227657 

.3529386 

.333735 

.3503793 

i 


I 

I 
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Table  8 


Population 

lOLO 

10L5 

lOLlO 

20L0 

20L5 

20L10 

30L0 

30L5 

30L10 


lOHO 

10H5 

lOHlO 

20H0 

20H5 

20H10 

30H0 

30H5 

30H10 


Statistical  Moments  of  RM4  Distributions 


Mean 

Var lance 

Std .  Dev. 

4188788 

.0941331 

.3068112 

4258803 

.0936967 

.306099 

423802 

.0897187 

.2995307 

3667525 

.0924979 

.3041346 

3749607 

.1118555 

. 334448 

3594583 

.1141094 

.3378008 

31271 

.109086 

. 3302818 

3256386 

.121957 

.3492234 

3049378 

.111348 

. 3336884 

4190774 

.0940824 

. 3067285 

425043 

.0851494 

.  2918037 

415741 

.0794311 

.281835 

3662538 

.111654 

. 3341467 

3620137 

.0943815 

. 3072156 

36832 

.1114834 

.3338913 

3221878 

.1245809 

. 35296 

3400995 

.1128749 

.  3359686 

3092027 

.1261533 

.3551807 

end  lx  D 


Sumnvary  Statistics  for  Research  Populations 


Table  9 

Summary  Statistics  for  Population:  lOLO 


EACl 

RMl 

Mean 

10552207. 

-  .  3596454 

Var lance 

1.004301e+14 

.  377067 

Std.  Dev. 

10021483 . 

.614058 

EAC2 

RM2 

Mean 

9693292 . 

-.4123724 

Variance 

2.789606e+13 

. 1024277 

Std.  Dev. 

5281672. 

. 320043 

EAC3 

RM3 

Mean 

9589969. 

-.418999 

Var  iance 

2.591912e+13 

.  094054 

Std.  Dev. 

5091082  . 

.  3066823 

EAC4 

RM4 

Mean 

9591994 . 

-  .4188788 

Var iance 

2.594234e+13 

.0941331 

Std.  Dev. 

5093362. 

.  3068112 

ACWP (60) 

Mean 

16558777. 

Var iance 

1 .693582e+12 

Std.  Dev. 

1301377. 
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Table  10 


Summary 

Statistics  for  Population: 

10L5 

gAg; 

RMl 

Mean 

10222382. 

- . 378712 

var lance 

1.917890e+14 

.6658007 

Std.  Dev. 

13848790. 

. 815966 

EAC2 

RM2 

Mean 

9484483 . 

- . 419743 

Var lance 

2 .765948e+13 

. 1021418 

Std.  Dev. 

5259228 . 

. 3195962 

EAC3 

RMl 

Mean 

9351947. 

- . 427719 

Var lance 

2.516910e+13 

.0930635 

Std.  Dev. 

5016882. 

.  305063 

EAC4 

RM4 

Mean 

9382043 . 

-  .4258803 

Var  lance 

2 . 534054e  +  13 

.  09  36967 

Std.  Dev. 

5033939 . 

.  306099 

ACWP (60) 

Mean 

16366193. 

Var lance 

1 . 483604e  +  12 

Std .  Dev . 

1218033 . 
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Table  11 


Summary 

Statistics  for  Population: 

lOLlO 

EACl 

RMl 

Mean 

10021643. 

-.393657 

var lance 

1.571839e+14 

.542814 

Std.  Dev. 

12537300. 

.736759 

EAC2 

RM2 

Mean 

9529091. 

-.419882 

Var  lance 

2 . 635388e  +  13 

.096849 

Std.  Dev. 

5133603. 

.  3112057 

EAC3 

RM3 

Mean 

9404131. 

-.4274625 

Var lance 

2.410610e+13 

.0884988 

Std.  Dev. 

4909795. 

.297489 

EAC4 

RM4 

Mean 

9464296. 

-  .  423802 

Var lance 

2 . 443815e+13 

.  0897187 

Std.  Dev. 

4943496. 

.  2995307 

ACWP (60) 

Mean 

16451551. 

Variance 

1.5l5712e+12 

Std.  Dev. 

1231143. 
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Table  12 

Sununary 

Statistics  for  Population: 

20L0 

EACl 

RMl 

Mean 

13815946. 

-.2258958 

Variance 

1.379802e+15 

5.26306 

Std.  Dev. 

37145684. 

2.294136 

EAC2 

RM2 

Mean 

11628031. 

-.36086 

Var lance 

3. 435702e  +  13 

.1002212 

Std.  Dev. 

5861486. 

. 3165773 

EAC3 

RM3 

Mean 

11520133. 

-.366683 

Var lance 

3.175133e+13 

.0925131 

Std.  Dev. 

5634831. 

. 3041596 

EAC4 

RM4 

Mean 

11519077 . 

-  .  3667525 

Var  lance 

3 . 175832e  +  13 

.0924979 

Std.  Dev. 

5635452. 

.  3041346 

ACWP160) 

Mean 

18191970. 

Variance 

1.957210e+12 

Std.  Dev. 

1399004. 
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Table  13 


w 

: 


i. 


Sunumary 

Statistics  for  Population: 

20L5 

EACl 

RMl 

Mean 

12113014. 

-.3279834 

Variance 

6 .668586e  +  14 

1.97177 

Std.  Dev. 

25823604. 

1.404197 

EAC2 

RM2 

Mean 

11279937. 

-.367753 

Var iance 

4 . 128770e  +  13 

.127554 

Std.  Dev. 

6425550. 

. 357147 

EAC3 

RM3 

Mean 

11087327. 

-.378269 

Variance 

3.572390e+13 

.  1106666 

Std.  Dev. 

5976948. 

.  3326658 

EAC4 

RM4 

Mean 

11146384 . 

-.3749607 

Var  iance 

3.610728e+13 

.  1118555 

Std.  Dev. 

6008933. 

.  334448 

ACWP{ 60) 

Mean 

17848886. 

Var lance 

1.769823e+12 

Std.  Dev. 

1330347. 
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Table  14 


Summary 

Statistics  for  Population: 

20L10 

EACl 

RMl 

Mean 

12857628. 

-.284222 

Variance 

2.633370e+14 

.834229 

Std.  Dev. 

16227658. 

.913361 

EAC2 

RM2 

Mean 

11533638. 

- .3589566 

Var lance 

4.092710e+13 

.1269422 

Std.  Dev. 

6397431. 

. 3562894 

EAC3 

RMl 

Mean 

11413822. 

-.3662427 

Var lance 

3.663916e+13 

.111564 

Std.  Dev. 

6053029. 

.334012 

Mean 
Var lance 
Std .  Dev. 

Mean 

Variance 


SAC  4 

11535967. 
3.747280e+13 
6121504 . 


ACWP(60 


18064804. 


1.984660e+12 


RM4 

-.3594583 
. 1141094 
.3378008 


Std .  Dev . 


1408780. 


Table  15 


Summary 

Statistics  for  Population: 

30L0 

EACl 

RMl 

Mean 

16645910. 

-.1369596 

Var lance 

2 .687206e  +  15 

8.68556 

Std.  Dev. 

51838260. 

2.947127 

EAC2 

RM2 

Mean 

13674372. 

-.3063355 

Var lance 

4.855310e+13 

. 1207366 

Std.  Dev. 

6968005. 

. 3474717 

EAC3 

RM3 

Mean 

13546857 . 

-.3126217 

Var lance 

4 . 3913146  +  13 

. 1091058 

Std.  Dev. 

6626699  . 

. 3303117 

EAC4 

RM4 

Mean 

13545404 . 

-  .  31271 

Var lance 

4 . 392570e  +  i3 

. 109086 

Std.  Dev. 

6627646 . 

. 3302818 

ACWP( 60 ) 

Mean 

19709986. 

Variance 

2.271806e+12 

Std.  Dev. 

1507251. 
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Table  16 


I 

V' 

«V« 


U 


m 


i 


iX 


Summary  Statistics  for  Population:  30L5 


Mean 


Variance 


Std.  Dev. 


Mean 


Var lance 


Std.  Dev. 


Mean 


Variance 


Std.  Dev, 


Mean 


Var  lance 


Std.  Dev. 


Mean 


Variance 


Std.  Dev, 


EACl 


16524732 


2.759790e+15 


52533704. 


SAC  2 


13362036. 


5. 388900e  +  13 


7340911. 


EAC3 


13200820, 


4.809944e+13 


6935378 


SAC  4 


13290375. 


4 .8787946  +  13 


6984836. 


ACWP( 60 


19714760. 


2.293340e+12 


1514378. 


-.1432385 


8.85594 


2.975894 


- . 3220447 


. 1353002 


. 367832 


.3301655 


.1202938 


.346834 


-  .  3256386 


. 121957 


. 3492234 


Table  17 


Summary 

Statistics  for  Population: 

30L10 

EACl 

RMl 

Mean 

16629110. 

-.1376086 

Var lance 

2.687208e+15 

8.68554 

Std.  Dev. 

51838284. 

2.947124 

BAG  2 

RM2 

Mean 

13646116. 

-.3075712 

Var lance 

4 .830540e  +  13 

.1201053 

Std.  Dev. 

6950208. 

.  346562 

BAG  3 

RM3 

Mean 

13516016. 

-.3139995 

Var lance 

4.365570e+13 

.1084363 

Std.  Dev. 

6607248. 

.3292967 

BAG  4 

RM4 

Mean 

13694937. 

-  .  3049378 

Var lance 

4 . 485260e  +  13 

. 111348 

Std.  Dev. 

6697207. 

.3336884 

AGWP(60) 

Mean 

19704774. 

Variance 

2.280552e+12 

Std.  Dev. 


1510150. 


Table  18 


Summary 

Statistics  for  Population: 

lOHO 

EACl 

RMl 

Mean 

311815616. 

-.3901554 

Var lance 

4.206544e+16 

. 1616249 

Std.  Dev. 

205098592. 

. 402026 

EAC2 

RM2 

Mean 

300046656. 

- . 413259 

Var lance 

2 .650512e  +  16 

. 1012668 

Std.  Dev. 

162804000. 

. 3182245 

EAC3 

RM3 

Mean 

297163000. 

-.4191976 

Var lance 

2.483380e+16 

.0940038 

Std.  Dev. 

157587408. 

. 3066003 

EAC4 

RM4 

Mean 

297225792. 

-.4190774 

Var lance 

2. 485590e  +  16 

. 0940824 

Std.  Dev. 

157657568. 

. 3067285 

ACWP( 60 ) 

Mean 

513188000. 

Variance 

1.443799e+15 

Std.  Dev. 


37997352 


Table  19 


Summary 

Statistics  for  Population: 

10H5 

EACl 

RMl 

Mean 

312075456. 

-.395343 

var lance 

5. 103280e+16 

.206625 

Std.  Dev. 

225904432. 

.45456 

EAC2 

RM2 

Mean 

299169000. 

-.4216004 

Var  lance 

2 . 443547e  +  l6 

. 0895257 

Std.  Dev. 

156318464. 

. 2992085 

EAC3 

RM3 

Mean 

296437280. 

-.4268747 

Var  lance 

2.314036e4-16 

.0845853 

Std.  Dev. 

152119536. 

.  2908354 

SAC  4 

RMl 

Mean 

297388384 . 

-.425043 

Var  lance 

2.329998e+16 

.0851494 

Std.  Dev. 

152643296 . 

.  2918037 

ACWP( 60 ) 

Mean 

517600000. 

Var lance 

1.411815e+15 

Std.  Dev. 


37574124 


Table  20 


Summary 

Statistics  for  Population: 

lOHlO 

EACl 

RMl 

Mean 

314948480. 

- . 3894853 

Var lance 

4.706270e+16 

. 1913368 

Std.  Dev. 

216939440. 

.4374206 

EAC2 

RM2 

Mean 

302756800. 

- . 4143805 

Var lance 

2 . 282570e  +  16 

. 0826822 

Std.  Dev. 

151081776. 

. 287545 

EAC3 

RM3 

Mean 

300169440. 

-.4193546 

Variance 

2.168520e+16 

.0784237 

Std.  Dev. 

147259000. 

.2800423 

EAC4 

RM4 

Mean 

302044000. 

-.415741 

Var  lance 

2 . 197182e  +  16 

. 0794311 

Std.  Dev. 

148229000. 

.  281835 

ACWP(60) 

Mean 

516755392. 

Var lance 

1. 428110e  +  15 

Std.  Dev. 


37790332. 
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Table  21 


Summary 

Statistics  for  Population: 

20H0 

EACl 

RMl 

Mean 

372786496. 

-.3314503 

Var lance 

8 . 399460e  +  16 

. 271928 

Std.  Dev. 

289818240. 

. 521467 

EAC2 

RM2 

Mean 

356882000. 

-  .  360192 

Var lance 

3.863110e+16 

.  1244955 

Std.  Dev. 

196548000. 

.  352839 

EAC3 

RM3 

Mean 

353678000. 

-.3664585 

var lance 

3.510360e+16 

. 1115042 

Std.  Dev. 

187359488. 

.3339224 

EAC4 

RM4 

Mean 

353794000. 

-  .  3662538 

Var lance 

3.515320e+16 

.  111654 

Std.  Dev. 

187491840. 

.  3341467 

ACWP(60) 

Mean 

559863000. 

Variance 

1.688717e+15 

Std.  Dev. 

41094000. 
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Table  22 


Summary 

Statistics  for  Population: 

20H5 

EACl 

RMl 

Mean 

379096000. 

- . 3258837 

Var lance 

1.036054e+17 

. 359676 

Std.  Dev. 

321878000. 

. 59973 

EAC2 

RM2 

Mean 

361350304 . 

- . 359221 

Var lance 

3 . 319137e  +  16 

. 1012998 

Std.  Dev. 

182185000. 

. 3182763 

EAC3 

RM3 

Mean 

357890000. 

- . 365283 

Var lance 

3.065988e+16 

.0934018 

Std.  Dev. 

175099600. 

. 305617 

EAC4 

RM4 

Mean 

359740544 . 

- . 3620137 

Var lance 

3.099398e+16 

. 0943815 

Std.  Dev. 

176051000 . 

. 3072156 

ACWP( 60 ) 

Mean 

563845000. 

Variance 

1.672740e+15 

Std.  Dev. 

40899156. 
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Table  23 


Summary  Statistics  for  Population: 


Mean 


Var  lance 


Std .  Dev. 


Mean 


Var lance 


Std.  Dev. 


Mean 


Variance 


Std.  Dev, 


Mean 


Var lance 


Std.  Dev 


Mean 


Var lance 


Std.  De V , 


~$7 


364959488. 


EACl 


1 . 444690e  +  17 


380090656 . 


EAC2 


351440000 . 


3 .8802806  +  16 


196984272 . 


EAC3 


345994000 . 


3.  393490e  +  16 


184214272 . 


EAC4 


349659000 . 


3 . 467092e+16 


186201280 


ACWP(60 


553945344 


1.513225e+15 


38900192. 
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-  .  343044 


.  4570483 


.6760535 


-  .  365342 


1245045 


. 352852 


-  .  3749374 


1091143 


3303246 


-  .  36832 


1114834 


3338913 


Table  24 


Summary 

Statistics  for  Population: 

30H0 

EACl 

RMl 

Mean 

426566656 . 

- . 29367 

Var lance 

2 . 365435e  +  17 

.64024 

Std .  Dev. 

486357000. 

. 80015 

EAC2 

RM2 

Mean 

414407712 . 

- . 3109253 

Var lance 

5 . 381430e  +  16 

.146631 

Std.  Dev. 

231979000 . 

.3829244 

EAC3 

RMl 

Mean 

407460736 . 

- .3222148 

Var lance 

4.561820e+16 

. 1245656 

Std.  Dev. 

213584192. 

. 3529386 

EAC4 

RM4 

Mean 

407476000 . 

- . 3221878 

Var lance 

4 . 56]  857e  +  16 

. 1245809 

Std.  Dev. 

213585000. 

. 35296 

ACWP (60) 

Mean 

601688000. 

Var lance 

1 . 770306e  +  15 

Std .  Dev . 

42075000. 

Table  25 


fgn 


I 


^3 


Suminary  Statistics  for  Population:  30H5 


Mean 


var lance 


Std.  Dev. 


Mean 


Var  lance 


Std.  Dev. 


Mean 


Var lance 


Std.  Dev, 


Mean 


Var  lance 


Std .  Dev , 


Mean 


Var lance 


Std  .  Dev . 


EACl 


416611520. 


1 . 871683e  +  17 


432629504. 


EAC2 


394192512 . 


4 . 644060e  +  16 


215500800 


EAC3 


389761000 


4 . 108870e  +  16 


202703472 . 


SAC  4 


392229000 . 


4  .  167227e  +  16 


204139000. 


ACWP ( 6  0 


593819000 


2. 124235e  +  15 


46089416 . 


-.297774 


. 565738 


.7521555 


-  .336999 


.1256891 


3545266 


-  .  344216 


1113792 


333735 


-  .  3400995 


1128749 


3359686 


Table  26 


Summary 

Statistics  for  Population: 

30H10 

EACl 

RMl 

Mean 

428850560. 

-.29059 

Variance 

2.361100e+17 

.638918 

Std.  Dev. 

485911520. 

. 799324 

EAC2 

RM2 

Mean 

416608000 . 

- . 3078258 

Var lance 

5.269840e+16 

.1438957 

Std.  Dev. 

229561312 . 

. 379336 

EAC3 

RM3 

Mean 

410118400. 

- . 3183693 

Var lance 

4 . 486460e  +  16 

. 1227657 

Std.  Dev. 

211812592 . 

. 3503793 

EACl 

RM4 

Mean 

415633000 . 

- . 3092027 

Var  Lance 

4 .609816e  +  16 

. 1261533 

Std.  Dev. 

214704800 . 

. 3551807 

ACVP (60) 

Mean 

602419000. 

Var lance 

1.757942e+15 
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This  thesis  examined  the  robustness  of  methods  used  to 
compute  estimates  at  completion  of  contract  costs.  The 
four  techniques  examined  were  a  subset  of  those  used  by  the 
Cost  Performance  Report  Analysis  computer  program  to  derive 
estimates  at  completion. 

Monte  Carlo  simulation  methods  were  used  to  generate 
the  required  data  values  for  18  simulated  populations 
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overrun  scenarios.  A  statistical  evaluation  of  the 
robustness  of  the  four  estimate  at  completion  techniques 
was  performed  using  samples  from  each  of  the  18  simulated 
populations. 

The  results  of  this  analysis  indicate  that,  for 
practical  purposes,  none  of  the  selected  estimate  at 
completion  techniques  are  robust  for  the  cost  and  schedule 
overrun  scenarios  that  were  investigated.  The  tendency  of 
each  of  the  estimate  at  completion  models  was  to 
underestimate  the  final  contract  cost.  The  estimate  at 
completion  models  incorporating  mathematical  smoothing 
techniques  (i.e.  weightings  or  moving  averages)  provided 
relatively  stable  results  across  all  of  the  populations; 
however,  they  were  unable  to  consistently  predict  a 
contract's  completion  cost  within  30  percent  of  the  actual 
value.  Of  the  four  methods  investigated,  the  method  based 
upon  the  Cost  Performance  Index  for  a  single  month  was  the 
most  unreliable  predictor  of  the  final  contract  cost. 
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